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Preface 


This  documented  briefing  is  an  assessment  of  the  United  States  Air  Force’s  (USAF’s)  Build¬ 
ing  Partnership  (BP)  Seminars.  It  identifies  options  for  improving  the  alignment  of  USAF’s 
international  programs  to  further  develop  USAF’s  Campaign  Support  Plan.  The  findings  from 
this  briefing  were  developed  using  the  RAND  security  cooperation  assessment  framework 
and  from  case  studies  of  two  Unified  Engagement  BP  Seminars  held  in  Sweden  (in  2009)  and 
Estonia  (in  2010). 

This  briefing  is  part  of  a  substantial  and  growing  body  of  RAND  Project  AIR  FORCE 
work  dealing  with  partnership  building.  The  study  was  sponsored  by  the  Director  of  Oper¬ 
ational  Planning,  Policy  and  Strategy,  Office  of  the  Deputy  Chief  of  Staff  for  Operations, 
Plans  and  Requirements,  Headquarters  USAF  (HQ  USAF/A5X),  in  coordination  with  its 
Regional  Plans  and  Issues  Division  (AF/A5XX),  Concepts,  Strategy,  and  Wargaming  Division 
(AF/A5XS),  and  Strategic  Plans  and  Policy  Division  (AF/A5XP).  The  research  was  conducted 
within  the  Strategy  and  Doctrine  Program  of  RAND  Project  AIR  FORCE  as  part  of  a  larger 
fiscal  year  2010  study  “Support  to  the  Air  Force  Campaign  Support  Plan  and  AF/A5X  Inter¬ 
national  Programs.”  It  builds  on  previous  RAND  Project  AIR  FORCE  work,  including  the 
following: 

•  Jennifer  D.  P.  Moroney,  Joe  Hogler,  Jefferson  P.  Marquis,  Christopher  Paul,  John  E. 
Peters,  and  Beth  Grill,  Developing  an  Assessment  Framework  for  U.S.  Air  Force  Building 
Partnerships  Programs,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-868-AF,  2010. 

•  Jefferson  P.  Marquis,  Joe  Hogler,  Jennifer  D.  P.  Moroney,  Michael  J.  Neumann,  Chris¬ 
topher  Paul,  John  E.  Peters,  Gregory  F.  Treverton,  and  Anny  Wong,  Adding  Value  to 
Air  Force  Management  Through  Building  Partnerships  Assessment,  Santa  Monica,  Calif.: 
RAND  Corporation,  TR-907-AF,  2010. 

•  Jennifer  D.  P.  Moroney,  Jefferson  P.  Marquis,  Cathryn  Quantic  Thurston,  and  Gregory 
F.  Treverton,  A  Framework  to  Assess  Programs  for  Building  Partnerships,  Santa  Monica, 
Calif.:  RAND  Corporation,  MG-863-OSD,  2009. 

•  Jennifer  D.  P.  Moroney,  Kim  Cragin,  Eric  Cons,  Beth  Grill,  John  E.  Peters,  and  Rachel 
M.  Swanger,  International  Cooperation  with  Partner  Air  Forces,  Santa  Monica,  Calif.: 
RAND  Corporation,  MG-790-AF,  2009. 
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Summary 


USAF’s  UE  BP  Seminars,  managed  by  its  wargaming  division  (AF/A5XS),  are  designed  to 
support  the  USAF’s  biannual  Title  10  wargame,  Unified  Engagement.  These  seminars  focus 
primarily  on  incorporating  the  insights  of  countries  that  do  not  formally  participate  in  the 
larger  wargame  by  eliciting  their  reaction  to  a  number  of  supporting  scenarios  through  table- 
top  exercises  during  the  seminars.  To  date,  BP  Seminars  have  taken  place  only  in  the  areas  of 
responsibility  of  the  U.S.  Air  Forces  in  Europe  and  U.S.  Pacific  Air  Forces. 

We  used  RAND’s  security  cooperation  assessment  framework  to  assess  these  seminars. 
This  framework  looks  at  six  specific  elements:  guidance,  programs,  authorities,  stakeholders, 
levels  of  analysis,  and  indicators  and  metrics  for  assessment.  With  it,  we  assessed  the  seminars 
held  in  Sweden  in  2009  and  in  Estonia  in  2010.  Both  seminars  successfully  strengthened  coop¬ 
eration  among  the  United  States  and  its  partners  in  the  Baltic  region.  However,  in  applying  the 
framework,  the  RAND  team  identified  several  ways  to  significantly  enhance  the  BP  Seminars: 

As  with  similar  programs,  the  BP  Seminars  could  benefit  from  the  institutionalization 
that  authoritative  documentation,  such  as  an  Air  Force  instruction,  would  provide. 

Establishing  clearer  links  to  combatant  command  guidance  would,  among  other  things, 
provide  measurable  objectives  for  each  seminar.  Such  objectives  are  critical  for  post-seminar 
assessments. 

Cost  reduction  is  possible  through  both  identifying  and  involving  other  potential  stake¬ 
holders  at  the  outset  of  planning  a  future  BP  Seminar. 

After- action  reports  would  be  highly  beneficial  for  capturing  seminar  insights,  honing  the 
relevance  and  value  of  future  events,  and  demonstrating  the  importance  and  effectiveness  of 
the  BP  Seminars  to  critical  stakeholders. 

Developing  and  implementing  a  follow-up  mechanism,  such  as  post-event  interviews  and 
participant  surveys,  would  be  invaluable  for  assessing  the  extent  to  which  an  event  met  its 
objectives,  necessary  areas  for  follow-up,  and  how  to  tailor  future  events. 

Most  of  these  suggestions  would  not  be  difficult  to  incorporate  into  the  BP  program 
architecture  if  introduced  early  stage  in  the  event-planning  process.  Implementing  them  will 
help  increase  the  program’s  visibility  and  effectiveness,  reduce  its  costs,  and  further  demon¬ 
strate  its  value  within  USAF  and  to  U.S.  partners  worldwide. 

Finally,  this  briefing  also  demonstrates  the  application  of  RAND’s  security  cooperation 
assessment  framework  and  further  affirms  its  utility  and  broad  applicability  to  partnership 
building  programs  managed  by  USAF. 
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Introduction,  Purpose,  and  Tasks 


Purpose:  Support  development  of  the  USAF  Campaign  Support  Plan 
and  improve  alignment  of  AF/A5X-managed  international 
programs 

Task  1 — Develop  concept  for  country  air,  space,  and  cyberspace 
plans  (AF/A5XW) 

Task  2 — Consider  options  for  improving  the  alignment  of  AF/A5X- 
managed  international  programs  (AF/A5XX,  AF/A5XS, 
AF/A5XP) 


This  annotated  briefing  summarizes  the  results  of  case  studies  developed  from  two  Unified 
Engagement  (UE)  Building  Partnerships  (BP)  Seminars,  held  in  Sweden  (in  2009)  and  Estonia 
(in  2010),  and  demonstrates  a  proof  of  concept  using  the  RAND  Corporation’s  security  coop¬ 
eration  assessment  framework.  Both  seminars  brought  members  of  the  air  forces  of  (in  alpha¬ 
betical  order)  Denmark,  Estonia,  Finland,  Iceland,  Latvia,  Lithuania,  Poland,  and  Sweden 
together  with  that  of  the  United  States.  These  seminars  were  designed  to  explore  ways  to 
increase  partnerships  among  these  countries.  We  applied  the  assessment  framework  to  further 
improve  the  utility  of  these  seminars. 

The  work  described  in  this  documented  briefing  is  part  of  a  broader  research  effort  to 
assist  the  Director  of  Operational  Planning,  Policy  and  Strategy,  Office  of  the  Deputy  Chief 
of  Staff  for  Operations,  Plans  and  Requirements,  Headquarters  U.S.  Air  Force  (AF/A5X)  with 
development  and  implementation  of  USAF’s  Campaign  Support  Plan.  This  briefing  addresses 
the  second  task,  whose  purpose  was  to  consider  options  for  enhancing  the  potential  of  BP 
Seminars  to  help  improve  the  alignment  of  the  international  programs  that  the  seminars’  three 
key  stakeholders  manage.  These  three  stakeholders  are  Regional  Plans  and  Issues  Division  (AF/ 
A5XX),  Concepts,  Strategy,  and  Wargaming  Division  (AF/A5XS),  and  Strategic  Plans  and 
Policy  Division  (AF/A5XP). 
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Task  2  Identifies  Options  for  Improving  Alignment 
of  AF/A5X-Managed  International  Programs 

•  Tailor  RAND's  assessment  framework  for  the  UE  BP  Seminars 

•  BP  Seminars  are  case  studies 

-  Sweden  2009 

-  Estonia  2010 


This  task  provides  a  proof  of  concept  for  assessing 
USAF-managed  partnership-building  programs, 
focusing  on  the  BP  Seminars. 


To  identify  options  for  improving  the  alignment  of  AF/A5X-managed  international  programs, 
we  first  tailored  RAND’s  assessment  framework  to  the  UE  BP  Seminars  and  then  used  the  BP 
Seminars  held  in  Sweden  and  Estonia  as  cases  studies. 
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Agenda 

•  Key  elements  of  the  assessment  framework 

-  Security  cooperation  assessment  framework 

-  Stakeholder  roles 

•  Assessing  the  BP  Seminars 

•  Options  for  improving  the  BP  Seminars 


First,  this  documented  briefing  looks  at  the  key  elements  of  RAND’s  security  cooperation 
assessment  framework.  This  includes  elements  of  the  framework,  why  USAF  should  assess  the 
BP  seminars  program,  guidance,  authorities,  assessment  levels,  and  assigning  stakeholders  to 
the  assessment  levels.  We  then  identify  and  explore  potential  stakeholder  roles  based  on  respon¬ 
sibilities  and  authorities  each  exercises  regularly. 
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RAND's  Security  Cooperation  Assessment 
Framework  Has  Six  Key  Elements 


Guidance 

Programs 

Stakeholders 

Authorities 

Five  levels  of  assessment 

Indicators  and  metrics  for  assessment 

The  following  slides  provide  an  overview  of 
each  of  the  key  elements. 


RAND’s  security  cooperation  assessment  framework  has  six  key  elements: 

•  guidance 

•  programs 

•  stakeholders 

•  authorities 

•  five  levels  of  assessment 

•  indicators  and  metrics  for  assessment. 

The  following  slides  explore  each  of  these  elements  the  context  of  the  UE  BP  Seminars. 


Assessing  the  U.S.  Air  Force  Unified  Engagement  Building  Partnerships  Seminars  5 


Why  Should  USAF  Assess  Its 
BP  Seminars  Program? 

•  The  OSD  Guidance  for  Employment  of  the  Force  (GEF)  requires  the  military 
departments  to  assess  the  performance  and  effectiveness  of  their  security 
cooperation  programs  annually 

•  Program-level  assessment 

-  allows  USAF  to  take  a  more  global  view  of  its  partnership-building  mission 

-  facilitates  decisions  about  continuing,  expanding,  or  cutting  programs  and 
resources  devoted  to  partnership-building 

-  informs  decisions  regarding  trade-offs  among  other  programs  USAF 
manages 

•  However,  USAF  currently  has  no  systematic  process  for  assessing  its 
partnership-building  programs 

Assessments  should  be  undertaken  specifically  to  inform  the 
decisions  policymakers  and  program  managers  must  make. 


Why  should  USAF  assess  its  partnership-building  programs?  The  Office  of  the  Secretary  of 
Defense’s  (OSD’s)  GEF  stipulates  annual  assessments  of  the  performance  and  effectiveness  of 
the  military  departments’  security  cooperation  programs.1  While  assessment  currently  occurs  as 
the  guidance  stipulates,  USAF  does  not  have  a  systematic  process  for  assessing  its  partnership¬ 
building  programs.  Such  a  systematic  assessment  would  provide  a  number  of  tangible  benefits 
for  policymakers  and  program  managers,  including  allowing  USAF  to  take  a  global,  strategic 
view  of  its  partnership-building  mission  rather  than  its  individual  components;  informing 
program  management  about  the  aspects  that  need  to  be  improved  and  expanded  or  to  be  cut; 
and  making  more-informed  decisions  when  weighing  trade-offs  with  respect  to  USAF’s  other 
programs  and  missions. 


1  Office  of  the  Secretary  of  Defense  (OSD),  Guidance  for  Employment  of  the  Force ,  Washington,  D.C.:  Department  of 
Defense,  2008,  not  available  to  the  general  public. 
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OSD  and  USAF  Guidance  Determines  High-Level 
Security  Cooperation  Objectives 


•  Win  the  long  war  against  terrorism 

•  Defeat  irregular  warfare  threats 

•  Ensure  regional  stability 

•  Influence  the  behavior  of  key  nations 


GEF  (2008) 


•  Establish,  sustain,  and  expand  mutually 
beneficial  global  partnerships 

•  Ensure  that  global  partners  have  the  capabilities 
and  capacity  to  provide  for  their  own  national 
security 

•  Establish  capacities  to  train,  advise,  and  assist 
foreign  air  forces  and  conduct  security 
cooperation  activities  using  USAF  airmen  having  the 
appropriate  language  and  cultural  skills 

•  Develop  and  enhance  partnership  interoperability, 
integration,  and  interdependence 


Air  Force 
Global 
Partnership 
Strategy 
(2008) 


DB605°6 


For  USAF’s  partnership-building  programs,  two  main  sources  of  guidance  determine  security 
cooperation  objectives: 

GEF  views  security  cooperation  as  a  campaign-level  mission  and  stipulates  a  number  of 
broad  high-level  goals  directly  relevant  to  security  cooperation.1  These  include  the  role  of  secu¬ 
rity  cooperation  as  it  relates  to  defeating  terrorism,  defeating  irregular  warfare  threats,  ensuring 
regional  stability,  and  influencing  the  behavior  of  key  nations. 

The  Air  Force  Global  Partnership  Strategy  is  more  specific,  citing  partnership-building 
as  the  centerpiece  of  USAF  security  cooperation.  As  a  result,  the  strategy  stipulates  a  number 
of  goals  for  partnership-building  programs,  including  developing  partnerships  that  are  estab¬ 
lished,  sustainable,  and  mutually  beneficial;  developing  global  partner  capabilities  and  capaci¬ 
ties  so  that  they  can  provide  for  their  own  national  security  needs;  ensuring  proper  USAF 
capacity  to  train,  advise,  and  assist  foreign  air  forces;  and  increasing  interoperability,  integra¬ 
tion,  and  interdependence  among  foreign  air  forces. 


1  Jennifer  D.  P.  Moroney,  Joe  Hogler,  Jefferson  P.  Marquis,  Christopher  Paul,  John  E.  Peters,  and  Beth  Grill,  Developing 
an  Assessment  Framework  for  US.  Air  Force  Building  Partnerships  Programs ,  Santa  Monica,  Calif.:  RAND  Corporation, 
MG-868-AF,  2010,  p.  10. 
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USAF  Has  Three  Types  of  Partnership-Building 
Programs  Under  Two  Main  Authorities 

•  USAF-managed  Title  10  programs 

-  Examples:  BP  Seminars,  Operator  Engagement  Talks, 

Military  Personnel  Exchange  Program 

•  Jointly  managed  Title  10  programs  that  USAF  executes 

-  Examples:  Chairman  of  the  Joint  Chiefs  of  Staff  exercises 

•  Department  of  State — managed  Title  22  programs  that 
USAF  implements 

-  Examples:  foreign  military  sales,  foreign  military 
financing,  international  military  education  and  training 

As  a  first  step,  USAF  should  focus  its  assessment 
on  the  programs  it  directly  manages. 


Two  titles  of  the  U.S.  Code  provide  general  authorization  for  USAF  partnership-building  pro¬ 
grams:  Title  10  authorizes  the  service  to  raise,  train,  and  maintain  military  forces,  and  Title  22 
authorizes  security  assistance  to  other  countries.1  Furthermore,  these  programs  fall  into  three 
management  categories:  Air  Force,  joint,  and  Department  of  State.  We  specifically  focused  on 
ways  to  implement  RAND’s  assessment  framework  to  a  partnership-building  program  under 
AF/A5X’s  direct  purview,  such  as  the  BP  Seminars.  The  assessment  framework  could  also  be 
expanded  to  include  another  AF/A5X-managed  program,  the  Operator  Engagement  Talks,  or 
a  SAF/IA— managed  program,  the  Military  Personnel  Exchange  Program,  among  many  others. 


i 


U.S.  Code  Title  10,  Armed  Forces,  as  of  2009;  U.S.  Code  Title  22,  Foreign  Relations  and  Intercourse,  as  of  2009. 
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AF/A5X  Manages  the  BP  Seminars,  but 
Authorities  and  Stakeholders  Are  Broader 

•  BP  Seminars  (seminar  and  tabletop  exercise) 

-  USAF  controls  funding 

-  USAF  decides  objectives 

•Governing  authorities 

-  Department  of  Defense  Directive  5230.20 

-  No  Air  Force  instruction  or  mission  statement 
currently  exists 

•  Stakeholders 

-  USAF:  SAF/IA,  AF/A3/5,  component  commands 

-  Joint:  combatant  commands 

-  Partner  countries 


While  AF/A5X  manages  the  BP  Seminars,  the  authorities  and  stakeholders  involved  include  a 
much  broader  range  of  organizations.  The  Air  Force  controls  the  funding  of  and  determines 
objectives  for  the  BP  Seminars  and  tabletop  exercises.  The  sole  governing  authority  for  the  BP 
Seminar  program  currently  is  Department  of  Defense  Directive  5230.20,  Visits  and  Assign¬ 
ments  of  Foreign  Nationals,  because  no  relevant  Air  Force  instruction  currently  exists.1  Broadly 
speaking,  stakeholders  include  the  Deputy  Under  Secretary  of  the  Air  Force,  International 
Affairs  (SAF/IA);  the  Air  Force  Deputy  Chief  of  Staff,  Operations,  Plans,  and  Requirements 
(AF/A3/5);  combatant  commands;  component  commands;  and  partner  countries. 


1  Instructions  exist  for  other  Air  Force  and  jointly  managed  programs.  One  example  is  Air  Force  Instruction  16-107,  Mili- 
tary  Personnel  Exchange  Program ,  Washington,  D.C.:  Department  of  the  Air  Force,  February  2,  2006.  A  joint  example  is 
Air  Force  Instruction  (Interservice)  16-105,  Joint  Security  Cooperation  Education  and  Training. ,  January  3,  2011. 
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AF/A5X  Should  Determine  USAF  Stakeholder 
Roles  for  the  BP  Seminars 


SAF/IA 

USAF  is  responsible 
for  overall 
assessment 

AF/A3/5 

Component 

commands 

i 

Decides  objectives  (outcome) 


Determines  need 
Controls  funding 
Decides  objectives  (outcome) 

Designs  activities  (outputs) 


Designs  activities  (outputs) 
Implements  activities 


DB605.9 


Establishing  and  understanding  the  different  stakeholder  roles  is  fundamental  for  carrying  out 
assessments.  Each  stakeholder’s  exact  role  is  determined  by  such  aspects  as  current  interests  and 
equities  and  the  need  to  ensure  objectivity  and  avoid  potential  conflicts  of  interest  when  assess¬ 
ing  both  program  accomplishments  and  potential  shortcomings.  To  this  end,  AF/A5X  should 
determine  specific  stakeholder  roles  for  program  assessment  of  the  BP  Seminars.  A  notional 
setup  is  that  both  SAF/IA  and  AF/A3/5  play  a  role  in  deciding  outcome  objectives.  AF/A3/5 
would  also  determine  need  and  control  funding  and,  in  conjunction  with  component  com¬ 
mands,  design  program  activities.  Along  with  codesigning  activities,  component  commands 
would  also  implement  program  activities. 
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RAND's  Framework  Has  Five 
Levels  of  Assessment 


5.  Cost  effectiveness 


4.  Outcomes  and  effects 


3.  Process  and  implementation 


2.  Design  and  theory 


1.  Need  for  program 


RAND’s  framework  has  five  levels  of  assessment: 

1.  the  need  for  the  program  and,  from  a  U.S.  government  perspective,  whether  the  pro¬ 
gram  fills  a  specific  gap  or  niche  or  solves  a  problem 

2.  how  key  stakeholders  develop  program  theory  and  design 

3.  all  the  processes  undertaken  for  a  particular  program 

4.  outcomes  and  effects 

5.  cost-effectiveness. 

For  BP  Seminars,  level  3  includes  preparing  for  the  seminars  (developing  seminar  objectives, 
tabletop  exercise  scenarios,  securing  resources  for  the  seminar,  and  resolving  problems). 
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First ,  Develop  Questions 

Relative  to  Assessment  Levels . . . 

Level 

Seminars  and  Tabletop  Exercises 

5 

Cost 

effectiveness 

•  Were  U.S.  costs  reasonable  relative  to 
outcomes? 

•  Did  the  United  States  pay  costs  that  the 
partner  country  should  have  paid? 

•  Were  there  process  expenses  that  did  not 
contribute  to  the  desired  outcomes? 

•  Can  the  same  outputs  and  outcomes  be 
generated  more  efficiently? 

•  What  are  the  primary  drivers  of  process 
costs?  Can  efficiencies  be  gained? 

4 

Outcomes 
and  effects 

•  Were  U.S.-partner  relationships 
strengthened? 

•  Were  avenues  for  future  interaction  opened? 

•  Were  foundations  laid  for  increasing 
shared  understanding? 

3 

Process 

and 

implementation 

•  Did  event  proceed  as  planned? 

•  Did  expected  attendees  participate? 

•  Did  participants  have  positive  interactions? 

•  Were  participants  satisfied  with  the 
interaction? 

•  Were  ranks  and  levels  effectively  matched 
by  all  participants? 

•  Were  necessary  inputs  provided? 

•  Were  program  specific  outputs  generated? 

2 

Design 
and  theory 

•  Did  the  shared  experiences  yield  positive 
relationship  benefits? 

•  What  kinds  of  workshops,  seminars,  and 
activities  yield  the  best  results? 

•  Is  the  frequency  of  these  events  adequate? 

•  Was  the  duration  of  the  event  adequate? 

•  Was  the  proper  balance  of  attendee 
expertise  achieved? 

•  Were  the  appropriate  levels/ranks  invited? 

1 

Need  for 
program 

•  Does  the  United  States  still  need  to  improve 
or  maintain  relations  with  chosen  partners? 

•  Did  the  measurable  benefit  derived  from 
this  event  support  strategic  end  states? 
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In  applying  the  RAND  framework’s  five  levels  of  assessment  to  the  BP  Seminars,  we  developed 
a  number  of  questions  for  each  assessment  level  to  build  indicators  and  other  means  of  assess¬ 
ing  the  BP  Seminars.  The  questions  above  are  illustrative  and  serve  as  a  starting  point  for  dis¬ 
cussion  among  the  various  stakeholders.  Using  these  or  more  event-tailored  questions  will  help 
elicit  important  ideas,  comments,  and  further  questions  that  stakeholders  may  have. 
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Having  identified  the  three  main  stakeholders  (AF/A3/5,  SAF/IA,  and  the  component  com¬ 
mand),  we  notionally  linked  them  to  the  assessment  levels  and  corresponding  questions  to 
designate  which  stakeholder  would  answer  particular  types  of  assessment  questions  to  be  used 
by  the  integrator  (in  this  instance,  SAF/IA)  to  carry  out  programmatic  assessments.  These 
notional  assignments  were  based  on  the  various  roles,  responsibilities,  and  authorities  these 
stakeholders  have  and  exercise  on  a  daily  basis.  This  linked  AF/A3/5  to  questions  on  the  need 
for  the  program  (level  1),  cost  effectiveness  (level  5),  and  design  and  theory  (level  2).  The  com¬ 
ponent  commands  were  linked  with  design  and  theory  (level  2)  and  process  and  implementa¬ 
tion  (level  3).  SAF/IA  was  linked  to  outcomes  and  effects  (level  4). 
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We  Identified  Four  Assessment 
Roles  for  Stakeholders 

•  Data  collector 

-  Gathers  and  aggregates  data  according  to  standards  of  the  assessor 
organization 

•  Assessor 

-  Sets  the  standards  and  methods  for  data  collection  and  evaluation 

•  Reviewer 

-  Ensures  that  assessors  have  set  appropriate  standards  and  evaluation 
methods  and  that  the  data  collector  is  adhering  to  them 

•  Integrator 

-  Organizes  and  synthesizes  programmatic  assessments  to  meet  OSD 
and  USAF  requirements 


Stakeholders  also  have  four  roles  during  an  assessment.  First,  the  data  collector  is  responsible 
for  collecting  and  aggregating  data  according  to  standards  set  by  the  assessor  organization. 
Because  all  stakeholders  collect  data,  all  are  also  data  collectors.  Those  that  control  resources  or 
conduct  administrative  and  executive  processes  are  the  core  data  collectors. 

Second,  the  assessor  sets  standards  for  the  collection  and  evaluation  of  data  for  the  vari¬ 
ous  data  collectors.  The  stakeholders  that  establish  need,  set  outcome  objectives,  or  set  output 
objectives  should  conduct  the  assessments.  AF/A5XX  establishes  need  and  objectives,  and  SAF/ 
IA  sets  outcomes  through  USAF’s  Global  Partnership  Strategy.  To  ensure  assessment  objectiv¬ 
ity  and  to  minimize  the  potential  for  possible  conflicts  of  interest,  it  is  very  important  that  the 
stakeholders  tasked  as  assessors  not  also  be  tasked  as  reviewers. 

Third,  the  reviewer  is  responsible  for  ensuring  that  the  assessors  have  set  appropriate  data 
collection  standards  and  developed  appropriate  evaluation  methods  and  that  data  are  being 
collected  and  evaluated  according  to  these  standards  and  methods.  Stakeholders  that  establish 
need  or  set  outcome  objectives  should  also  conduct  reviews. 

Finally,  the  integrator  is  responsible  for  organizing  and  synthesizing  programmatic  assess¬ 
ments  to  meet  OSD  and  Air  Force  requirements.  Stakeholders  that  establish  need  or  set  out¬ 
come  objectives  should  integrate  assessments. 
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Notional  Unking  of  Stakeholders  to  Assessment 
Roles  for  the  BP  Seminar  Program 


Assessment 

Decision 

Data 

Collectors 

Assessors 

Reviewers 

Integrator 

Need 

AF/A3/5 

Component 

commands 

AF/A3/5 

SAF/IA 

AF/A3/5 

SAF/IA 

Design  and  theory 

AF/A3/5 

Component 

commands 

AF/A3/5 

Component 

commands 

SAF/IA 

AF/A3/5 

SAF/IA 

Process  and 
implementation 

AF/A3/5 

Component 

commands 

AF/A3/5 

SAF/IA 

AF/A3/5 

SAF/IA 

Outcomes  and 
effects 

AF/A3/5 

Component 

commands 

AF/A3/5 

Component 

commands 

SAF/IA 

SAF/IA 

Cost- 

effectiveness 

AF/A3/5 

Component 

commands 

AF/A3/5 

Component 

commands 

SAF/IA 

SAF/IA 
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This  slide  notionally  links  assessment  roles  to  various  stakeholders.  This  configuration  would 
change  using  different  program  examples,  but  for  such  Air  Force-managed  programs  as  the  BP 
Seminars,  SAF/IA  would  fill  the  integrator  role  because  ir  is  responsible  for  the  strategic  guid¬ 
ance  and  setting  the  outcome  objectives.  AF/A3/5  and  the  component  commands  also  have 
critical  roles  to  play,  primarily  as  data  collectors  and  assessors. 
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Agenda 

•  Elements  of  the  assessment  framework 

•  Assessing  the  BP  Seminars 

-  Source  review  and  key  interviews 

-  Need  for  a  clear  mission  statement 

-  Case  studies  and  findings 

■  Sweden  (2009) 

■  Estonia  (2010) 

•  Options  for  improving  the  BP  Seminars 


We  will  now  turn  from  our  discussion  of  RAND’s  assessment  framework  and  its  general  appli¬ 
cation  and  focus  on  or  assessment  of  the  BP  Seminars.  We  begin  by  looking  at  the  results  of  an 
extensive  source  review  and  focusing  discussion  with  key  U.S.  and  partner-country  stakehold¬ 
ers  we  conducted  early  in  the  project.  Next,  recognizing  a  need  for  consistent  application  and 
purpose,  we  present  a  preliminary  mission  statement  for  the  BP  Seminars.  Finally,  we  look  at 
the  objectives  and  results  of  our  two  case  studies,  Sweden  and  Estonia. 
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We  Reviewed  Relevant  BP  Seminar  Reports  and 

Interviewed  Key  Stakeholders 

•  BP  Seminar  reports 
-  India  2007 

•  Key  partner-country  stakeholders 
(Sweden  and  Estonia  events) 

-  Romania  2008 

-  Sweden  2009 

-  Host-nation  officials  directly 
involved  with  planning  and 
execution 

-  Other  nations  attending  the 
events 

•  Key  U.S.  stakeholders 

-  AF/A5XS  staff 

-  USAFE  Operations  and 
Strategy  and  Plans 

-  PACAF  Strategy  and 

Plans 
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We  undertook  a  review  of  recent  BP  Seminar  reports,  including  those  for  the  seminars  in 
India,  Romania,  and  Sweden,  then  interviewed  representatives  of  key  stakeholders.  Key  U.S. 
interviewees  represented  AF/A5XS,  U.S.  Air  Forces  in  Europe  (USAFE)  Operations  (A3)  and 
Strategy  and  Plans  (A5),  and  Pacific  Air  Forces  (PACAF)  Strategy  and  Plans  (A5).  Key  partner- 
country  stakeholder  interviewees  included  host-country  officials  from  Sweden  and  Estonia  and 
officials  from  other  countries  attending  the  seminars. 
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Discussions  with  U.S.  Stakeholders 
Revealed  Common  Challenges 

•  BP  Seminar  format  differs  between  the  two  main  AORs 

-  USAFE  mainly  supports  problem-solving,  multilateral  events 

-  PACAF  favors  a  bilateral  approach 

•  Planning  is  difficult  because  of  the  uncertainty  about  the  number  of 
events  in  each  AOR  each  year,  especially  for  PACAF 

-  A  set  number  of  events  is  needed  for  each  year 

•  Lack  of  dedicated  seminar  resources  is  a  recurring  concern 

-  Obtaining  additional  resources  from  COCOMs  and  MAJCOMs  is 
challenging  and  uncertain 

-  It  is  often  difficult  to  get  joint  organizations  to  participate  because 
this  is  perceived  as  a  service-level  program 

•  After-action  reports 

-  These  can  be  approved  by  Headquarters  USAF 

-  These  should  be  briefed  to  MAJCOM  commanders 


Discussions  with  U.S.  stakeholders  revealed  common  challenges  across  the  two  main  areas 
of  responsibility  (AORs)  in  which  the  BP  Seminars  take  place:  U.S.  European  Command 
and  the  U.S.  Pacific  Command  AOR.  BP  Seminars  in  the  U.S.  European  Command  AOR 
involve  large,  multilateral  events,  involving  many  partner  countries,  as  with  the  two  case  stud¬ 
ies.  Seminars  in  the  U.S.  Pacific  Command  AOR,  however,  are  primarily  bilateral  (one  partner 
country  each). 

These  differences  reflect  the  broader  alliance  structures  in  the  two  AORs.  The  North 
Atlantic  Treaty  Organization  (NATO)  in  Europe  is  multilateral,  while  the  United  States  main¬ 
tains  “hub-and-spoke”  treaty  alliances  with  many  of  its  partner  countries  in  Asia. 

Planning  for  BP  Seminars  can  be  a  challenge  in  any  case,  especially  for  PACAF,  because 
of  the  uncertainty  surrounding  the  total  number  of  events  each  year.  Setting  a  specific  number 
annually  would  dramatically  aid  planning  of  future  seminars.  Furthermore,  dedicated  seminar 
resources  are  lacking,  and  seeking  the  necessary  resources  from  the  combatant  commands  and 
the  major  commands  (MAJCOMs)  continues  to  be  challenging  and  uncertain.  Our  research 
also  revealed  a  perception  that  the  seminars  are  focused  on  air  force  issues  for  the  United  States 
and  the  partner  countries,  which  contributes  to  the  challenge  of  soliciting  participation  from 
other  services. 
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U.S.  and  Partner-Country  Stakeholders 
Cited  a  Variety  of  BP  Seminar  Objectives 

The  objectives  focused  on  building  relationships  and  capabilities: 


•  Relationship  building  and 
establishing  new  contacts 

•  Expose  foreign  air  forces  to  USAF 
mindsets 

•  Reinforce  USAF  commitment  to 
partners 

•  Encourage  countries  to  "take 
ownership" 

•  Promote  regional  stability 

•  Provide  a  scenario-driven  discussion 
designed  to  open  up  avenues  of 
cooperation 

•  Engage  non-UE  partners 
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Open  avenues  of  cooperation  to 
building  capacity 
Promote  humanitarian  assistance 
and  disaster  relief  capabilities 
Set  the  stage  for  future 
interoperability 

Engage  partner  countries  outside 
the  U.S.,  United  Kingdom, 

Canada,  Australia,  and  New 
Zealand  intelligence-sharing 
partnership  to  discuss  air,  space, 
and  cyberspace  capabilities 
Employ  different  kinds  of  air, 
space,  and  cyberspace  capabilities 


The  focused  discussions  with  U.S.  and  partner-country  stakeholders  elicited  quite  a  few  BP 
Seminar  objectives,  including  relationship  building;  reinforcing  USAF  partnership  commit¬ 
ments;  promoting  regional  stability;  engaging  non-UE  partners;  engaging  partners  outside  the 
U.S.,  United  Kingdom,  Canada,  Australia,  and  New  Zealand  intelligence  sharing  partnership 
to  discuss  air,  space,  and  cyberspace  capabilities;  and  promoting  humanitarian  assistance  and 
disaster  relief  capabilities.1  The  number  and  diversity  of  objectives  listed  demonstrates  the  need 
for  a  unifying  mission  statement. 


1  This  intelligence-sharing  partnership  is  sometimes  referred  to  as  Five-Eyes,  denoting  the  five  participants  involved,  as 
listed  above.  For  further  information  on  this  partnership,  see  Martin  Rudner,  “Hunters  and  Gatherers:  The  Intelligence 
Coalition  Against  Islamic  Terrorism,”  International  Journal  of  Intelligence  and  Counterintelligence ,  Vol.  17,  No.  2,  2004.  pp. 
193-230. 
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A  Clear  Mission  Statement  Will  Help  Ensure 
Consistent,  Effective  Seminar  Implementation 

•  Meaningful  assessment  of  BP  Seminars  requires  a  clear  mission  statement 
and  measureable  objectives 

•  We  drafted  a  preliminary  mission  statement  that 

-  captures  common  themes  from  discussions 

-  addresses  links  to  strategic  guidance 

-  highlights  the  "niche"  approach  of  the  BP  Seminars 

•  Proposed  mission  statement: 

Building  Partnerships  Seminars  are  designed  to  develop 
and  maintain  mutually  beneficial  relationships  with 
allies  and  partners  through  a  problem-solving  approach 
to  exploring  opportunities  for  cooperation  on  the 
bilateral  and  multilateral  levels. 


We  propose  that  AF/A5X  develop  a  clear  mission  statement  for  the  program  to  ensure  consis¬ 
tent  and  effective  implementation  of  future  BP  Seminars.  Such  a  mission  statement  is  impor¬ 
tant  for  bringing  various  stakeholders  into  a  common  understanding  of  the  BP  Seminars, 
something  that  may  currently  be  lacking.  Along  with  measurable  objectives,  a  clear  mission 
statement  would  also  aid  meaningful  evaluation  by  providing  for  an  apples-to-apples  compari¬ 
son  of  multiple  seminars  over  time. 

The  proposed  mission  statement  is  the  result  of  common  themes  from  seminar  discus¬ 
sions,  highlighting  the  niche  and  gap-filling  approach  of  the  BP  Seminars. 
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The  UE  BP  Seminar  in  Sweden 
Had  Three  Key  Objectives 

1.  Strengthen  air  staff  relationships 

2.  Anticipate  future  airpower  challenges 

3.  Address  RSAF  security  interests 

The  Swedish  BP  Seminar  assessment  focused  on 
outputs  and  outcomes  because  significant  time 
had  passed  since  the  last  event. 


For  the  first  case  study,  we  looked  at  a  BP  Seminar  held  in  Sweden  in  2009.  We  examined  in 
detail  the  seminar’s  three  objectives,  which  USAF  and  the  Royal  Swedish  Air  Force  (RSAF) 
had  developed  jointly,  and  the  seminar  results: 

•  strengthen  air  staff  relationships 

•  anticipate  future  air  power  challenges 

•  address  RSAF  security  interests. 

Because  significant  time  had  passed  since  the  previous  seminar,  this  assessment  focused  on  this 
event’s  outputs  and  outcomes. 


Assessing  the  U.S.  Air  Force  Unified  Engagement  Building  Partnerships  Seminars 


21 


Sweden  BP  Objective  1: 
Strengthen  Air  Staff  Relationships 


Objective: 

Strengthen  relationships  among  air 
leaders  and  staffs  of  key  regional  allies 
and  partners 

•  Subsequent  events  planned  including 
the  same  leaders  and  staffs  from  the 
participating  countries 

•  Invitations  issued  to  leaders 

and  staffs  to  attend  events  RSAF  - 
hosted 

•  USAF  responded  to  follow-up  (or 
other)  requests  for  information, 
etc. 


Results: 

Not  difficult  to  make  connections  for 
follow-up  activities  because  of  the  dynamic 
USAF-RSAF  relationship  at  senior  and 
working  levels 

•  Information  requested  on  Eagle  Vision  and 
the  subsequent  RSAF  visit  to  the  United 
States  can  be  tied  to  discussions  held  in 
the  context  of  the  BP  Seminar,  and  is  an 
outcome 

•  An  RSAF  white  paper  looked  at  issues 
raised  at  the  BP  Seminar  on  the  High  North 
and  is  also  an  outcome 

•  First  USAF-RSAF  joint  exercises  were 
planned  for  August  2010 
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The  first  objective  of  the  BP  Seminar  in  Sweden  was  to  strengthen  relationships  among  air  staff 
leaders  and  the  staffs  of  key  regional  allies  and  partners.1  Subsequent  events  were  planned  to 
reengage  the  same  leaders  and  staffs  from  the  participating  countries.  For  example,  RSAF  took 
the  initiative  to  invite  the  seminar  participants  to  attend  a  number  of  events  they  were  hosting. 
Lastly,  follow-up  requests  for  information  were  provided  by  USAF. 

Making  connections  to  follow-up  activities  is  not  difficult  because  of  the  dynamic  rela¬ 
tionship  between  USAF  and  RSAF  at  both  senior  and  working  levels.  Outcomes  from  the 
seminar  included  RSAF’s  interest  in  Eagle  Vision,  “a  deployable  ground  station  for  processing 
imagery  received  directly  from  commercial  satellite  platforms,”  and  a  subsequent  visit  to  the 
United  States.2 


1  In  addition  to  the  United  States  and  Sweden,  other  partner-country  attendees  at  this  seminar  included  Denmark,  Esto¬ 
nia,  Finland,  Iceland,  Latvia,  Lithuania,  and  Poland. 

2  Federation  of  American  Scientists,  “Eagle  Vision,  Eagle  Vision  II,”  Washington,  D.C.,  November  21,  1998. 
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Sweden  BP  Objective  2: 
Anticipate  Future  Airpower  Challenges 


Objective: 

Anticipate  future  airpower  challenges  and 
advance  operational  concepts  for  future 
capabilities,  cooperation,  and 
interoperability,  particularly  in  Baltic 
security 

•  Scenario  sufficiently  addressed  this  topic 

•  Regional  CONOPs  discussed  and 
developed,  possibly  with  support  of  the 
United  States,  another  ally,  NATO,  or  the 
European  Union 

•  Regional  workshops,  tabletop  exercises, 
or  field  training  exercises  held  to  test 
CONOPs  at  a  lower  level 

•  Baltic  countries  push  their  security 
concepts  in  NATO  forums 

•  Other  evidence  of  deepening  regional 
relations  on  Baltic  security 
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Results: 

•  Scenarios  focused  well  on  the  three 
areas,  but  these  topics  were  not  new  to 
Sweden 

•  Working  through  these  topics  promoted 
shared  understanding  that  was  valuable 
for  better  understanding  each  partner's 
logic 

•  Again,  it  is  difficult  to  tie  any  results 
back  to  UE  because  of  the  dynamic 
relationship  between  the  two  air  forces 

•  RSAF  did  not  follow  up  directly  with 
any  of  the  partner  nations 

•  RSAF  did  not  include  High  North 
scenario  in  air  force  training  program; 
focus  remains  on  Baltic  security 


The  second  objective  was  anticipating  future  airpower  challenges  and  advancing  operational 
concepts  for  future  capabilities,  cooperation,  and  interoperability.  Not  surprisingly,  the  scenario 
for  the  tabletop  exercise  focused  on  future  challenges  and  operational  concepts  relevant  specifi¬ 
cally  to  Baltic  security.  While  these  topics  were  not  new  to  RSAF,  the  shared  understanding 
attained  with  other  partner  nations  was  highly  beneficial,  as  were  the  insights  about  specific 
issues  and  challenges  they  collectively  face.  Because  this  scenario  was  held  at  a  high  (strategic) 
level,  among  the  air  staffs  of  the  attending  countries,  follow-on  regional  workshops,  table- 
top  exercises,  and  field  exercises  were  held  to  test  the  newly  developed  concepts  of  operation 
(CONOPs)  at  the  lower  operational  and  tactical  levels.  A  further  benefit  this  objective  yielded 
was  that  the  Baltic  countries  have  strengthened  and  deepened  their  relations  and  can  now 
better  promote  Baltic  security  issues  in  NATO  forums.  Unfortunately,  from  a  sustainability 
standpoint,  RSAF  did  not  follow  up  directly  with  any  of  the  partner  nations.  Also,  RSAF  has 
yet  to  include  a  Fligh  North  scenario  focused  on  Arctic  security  issues  into  its  own  training 
program.1 


1  High  North  scenarios  focus  on  a  variety  of  Arctic  security  issues  and  concerns  (e.g.,  access  to  energy  reserves)  shared  by 
U.S.  NATO  allies  and  Sweden. 
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Sweden  BP  Objective  3: 

Address  Swedish  Air  Force  Security  Interests 


Objective: 

Address  RSAF  interests  (i.e.,  Baltic  region 
security.  High  North,  stability,  and 
counterpiracy  operations) 

•  Scenario  sufficiently  addressed  these 
topics 

•  Follow-on  events  planned  or  held  or 
actions  taken  in  partner  nation  that  link  to 
these  three  focus  areas  and  build  on 
information  acquired  during  the  event 

•  Data  acquired  used  to  inform  new 
Swedish  strategies  or  doctrine 


Results: 

•  These  were  both  Swedish  and  U.S. 
objectives.  Sweden  suggested  objectives 
it  felt  the  United  States  would  support 

•  The  fledgling  Nordic  Defence  Cooperation 
organization  is  a  possible  forum  in  which 
to  discuss  these  issues.  This  is  a  series  of 
meetings  of  the  Nordic  countries, 
including  at  headquarters  level,  but  also 
includes  exchange  officers.  To  date,  ideas 
from  UE  not  presented  here 

•  One  outcome  from  the  partner  countries: 
Lithuania  distributed  materials  from  the 
event  to  its  Ministry  of  Defence  staff, 
then  held  a  seminar  to  discuss  the 
scenarios 
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The  final  objective  was  to  address  RSAF  interests.  Sweden  suggested  Baltic  regional  security, 
Fligh  North  issues,  stability,  and  counterpiracy  operations,  believing  that  the  United  States 
would  be  most  interested  and  supportive  of  these  but  not  necessarily  because  all  (i.e.  High 
North)  were  of  specific  interest,  at  least  initially.  All  these  objectives  were  successfully  incorpo¬ 
rated  into  the  scenario. 

While  the  Nordic  Defense  Cooperation  organization,  created  in  late  2009  by  Nordic 
Council  member  states,  provides  an  outlet  for  regular  interaction  between  the  air  staffs  of  Den¬ 
mark,  Finland,  Iceland,  Norway,  and  Sweden  on  these  topics,  membership  in  the  organization 
does  not  extend  to  a  number  of  other  Baltic  states,  including  seminar  participants  Estonia, 
Latvia,  Lithuania,  and  Poland.1  As  a  result,  the  BP  Seminar  provided  a  valuable  forum  to  dis¬ 
cuss  common  issues  with  the  Baltic  states,  an  opportunity  that  otherwise  has  heretofore  not 
existed. 

The  results  of  this  objective  were  immediately  apparent.  Sweden  used  seminar  data  to 
update  strategies  and  doctrine,  and  Lithuania  distributed  materials  from  the  event  to  its  Min¬ 
istry  of  Defense  and  held  a  seminar  to  discuss  the  scenario. 


1  The  Nordic  Council  also  includes  the  autonomous  territories  of  Greenland,  Aland,  and  the  Faroe  Islands  and  has  offices 
in  nonmember  countries  of  Estonia,  Latvia,  and  Lithuania  as  well  as  two  in  Russia.  See  Nordic  Council,  “Countries  and 
Territories,”  Norden  website,  undated. 
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UE  BP  Seminar  Sweden: 

Participants'  Comments 

•  Denmark 

•  Sweden 

-  Will  recommend  that  the  High  North 

-  Will  incorporate  a  High  North 

be  included  in  its  defense  planning 

scenario  and  lessons  from  the  BP 

scenarios 

Seminar  into  RSAF  training  program 

•  Finland 

-  Realized  that  Swedish  transformation 

-  Will  brief  the  Estonian  chief  of  defense 

planning  over  the  last  five  years  had 

on  the  High  North  scenario  from  the 

gotten  away  from  incorporating 

BP  Seminar 

Russia,  necessitating  change3 

*  Latvia  and  Lithuania 

-  Event  gave  them  a  chance  to  think 

strategically — they  do  not  have  many 

opportunities  to  do  this 

3  It  is  not  clear  that  the  BP  Seminar  caused  this  realization,  but  it  was  a  contributing  factor. 
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Comments  from  seminar  participants  highlighted  the  value  of  the  BP  Seminars  for  both 
building  relationships  and  sharing  information.  This  seminar  helped  Denmark,  Finland,  and 
Sweden  recognize  a  greater  need  to  focus  on  High  North  issues,  and  Denmark  and  Sweden 
specifically  announced  that  they  would  increase  their  focus  on  this  issue  and  would  implement 
the  insights  in  training  and  planning  scenarios.  Finland,  Latvia,  and  Lithuania  appreciated 
this  seminar  as  a  forum  for  discussing  future  strategic  issues — something  that  they  do  not  have 
many  opportunities  to  do.  USAF  benefited  from  gaining  a  deeper  understanding  of  current 
and  potential  future  regional  security  issues  in  the  High  North  and  from  further  developing 
relationships  with  its  Northern  European  partners.  These  insights  and  others  underline  the 
need  for  data  collection  to  both  measure  the  outcomes  of  future  seminars  and  increase  the 
effectiveness  of  future  events. 
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The  UE  BP  Seminar  in 
Estonia  Had  Three  Key  Objectives 

1.  Regional  crisis  support 

2.  Countering  hostile  influence  operations 

3.  Cooperative  ISR  operations 

Because  the  seminar  in  Estonia  had  just  occurred,  its 
assessment  focused  on  identifying  immediate 
outputs. 


For  the  second  case  study,  we  looked  at  a  BP  Seminar  held  in  Estonia  in  2010  by  taking  a 
detailed  look  at  the  seminar’s  three  objectives  and  their  results: 

1.  understanding  regional  crisis  support 

2.  countering  hostile  influence  operations 

3.  promoting  cooperative  intelligence,  surveillance,  and  reconnaissance  (ISR)  operations. 

Because  significant  time  had  passed  since  the  previous  seminar,  this  assessment  focused  on  this 
event’s  immediate  outputs. 
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Estonia  BP  Objective  1 
Regional  Crisis  Support 


Objective: 

Advance  allies'  and  partners'  thinking 

about  how  to  support,  and  receive  support 

from,  each  other  in  regional  crisis  situations 

•  Scenario  focused  attention  on  this  topic, 
provided  sufficient  time  to  discuss  options 

•  Partners  asked  questions  and/or  offer 
observations  based  on  their  own  national 
crisis  response  plans 

•  Partners  demonstrated  understanding  of 
the  importance  of  working  in  an 
interagency  and  regional  context 

•  Discussion  included  ways  to  train  and 
exercise  together 

•  Discussion  also  included  legal  rules  on  use 
of  force 


Results: 

■  At  a  very  high  level,  this  was  discussed 
in  one  breakout  session.  No  details  on 
how  this  might  work  in  practice 

■  Partner-country  national  response  plans 
were  not  discussed 

■  Working  together  in  a  regional  but  not 
interagency  context  was  discussed  in 
the  plenary  and  breakout  sessions 

■  Ways  to  train  and  exercise  together 
were  discussed  in  one  breakout  session, 
focused  on  Baltic  Operations  Field 
Training  Exercise 

■  Legal  rules  on  the  use  of  force  were 
briefed  in  plenary,  but  it  would  have 
been  better  to  have  this  presentation 
earlier  to  inform  discussions  in  move  1 
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The  first  objective,  advancing  allies’  and  partners’  thinking  on  how  to  support  and  receive 
support  from  each  other  during  regional  crisis  situations,  was  new  and  somewhat  novel  to  the 
participants.  Therefore,  discussions  based  on  the  scenario  set  the  foundation  for  further  explo¬ 
ration  and,  it  was  hoped,  eventual  realization  of  true  cooperation  and  support  during  crisis 
situations  among  Baltic  nations. 

Seminar  participants  shared  observations  on  their  respective  national  crisis  plans,  dis¬ 
cussed  ways  to  train  and  exercise  together,  and  discussed  the  various  rules  of  engagement  on 
the  use  of  force  that  the  participants  had  to  deal  with  in  responding  to  crisis  situations.  As  a 
preliminary  exploration  toward  future  cooperation  and  support,  the  discussion  remained  at  a 
high  level  and,  as  such,  national  response  plans  were  not  investigated. 
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Estonia  BP  Objective  2: 
Countering  Hostile  Influence  Operations 


Objective: 

Stimulate  deeper  regional  and  U.S. 
thinking  about  how  to  deter  and 
counter  future  hostile  operations  that 
increase  influence  (as  opposed  to 
outright  military  attacks) 

•  Scenario  identified  a  range  of  options 
on  this  topic  and  provided  sufficient 
time  to  discuss 

■  Influence  operations  concept  was 
addressed  in  the  discussion  so  that  all 
participants  fully  understood 


Results: 

The  range  of  options  was  discussed 
during  the  breakout  sessions,  but  the 
emphasis  was  mostly  on  the  capabilities 
required 

While  influence  operations  were 
discussed  in  the  breakout  sessions,  the 
concept  was  not  defined  with  any  degree 
of  specificity 
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The  second  objective  was  to  stimulate  regional  and  U.S.  thinking  about  how  to  deter  or  coun¬ 
ter  future  hostile  influence  operations,  such  as  the  cyber  attacks  against  Estonia  in  2007.  The 
scenario  conceptually  unpacked  the  meaning  of  influence  operations  to  provide  a  common 
understanding  for  discussion.  However,  the  concept  of  influence  operations  has  itself  not  yet 
been  defined  with  any  degree  of  specificity,  which  hampers  a  common  response. 

The  scenario  also  identified  a  range  of  potential  options  in  dealing  with  such  hostile 
operations,  and  a  breakout  session  discussion  focused  on  the  capabilities  required  to  deter  or 
repel  such  attacks. 
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Estonia  BP  Objective  3: 
Cooperative  ISR  Operations 


Objective: 

Promote  and  strengthen  concepts  for 
cooperative  ISR  operations  in  the  Baltic 
region,  including  for  energy  security  (in 
this  case,  regional  pipeline  security) 

•  Scenario  focused  attention  on  ISR 
operations  and  provided  sufficient  time 
to  discuss 

•  Range  of  regional  options  identified  to 
strengthen  pipeline  security 

•  Partners  requested  additional 
information 


Results: 

•  ISR  was  discussed  in  the  breakout 
session  in  the  context  of  support  to 
pipeline  surveillance 

•  Regional  options  for  strengthening 
pipeline  security  were  discussed 
during  breakout  session 

•  Danes  asked  for  further  information 
on  USAF  use  of  space-based 
capabilities,  particularly  satellites 
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The  final  objective  of  the  Estonia  seminar  was  promoting  and  strengthening  cooperative  ISR 
operations  in  the  Baltic,  including  energy  security.  The  scenario  specifically  looked  at  pipeline 
security,  and  the  breakout  sessions  focused  on  pipeline  surveillance  and  regional  operations  to 
strengthen  pipeline  security.  Partners  requested  additional  information  related  to  cooperative 
ISR  operations;  in  particular,  the  participants  from  Denmark  asked  for  further  information  on 
USAF  use  of  space-based  capabilities. 
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UE  BP  Seminar  Estonia: 
Participants'  Comments 


•  Denmark 

-  Plans  to  follow  up  with  USAFE  officials 
on  training  for  expeditionary  forces 
(understanding  CONOPS,  contrasting  the 
United  States  and  NATO) 

•  Estonia 

-  Information  from  the  scenario  to  be  used 
in  an  upcoming  Estonian  joint  exercise 
that  is  to  rehearse  the  Estonia  2018 
Defense  Strategy 


Lithuania 

-  Event  helped  to  anticipate  future 
challenges  in  the  Baltic  Sea 

Poland 

-  Report  to  be  briefed  to  chief  of 
training  in  Polish  Air  Force;  may 
use  insights  in  Polish  Air  Force 
training  and  exercises 

Sweden 

-  Event  moved  well  beyond  Sweden 
event  because  discussions  were 
more  open 


DB605=29 


Comments  from  participants  in  the  Estonia  seminar  again  highlighted  the  value  of  the  BP 
Seminars  and  underlined  the  need  for  data  collection  to  both  measure  the  outcomes  of  future 
seminars  and  increase  the  effectiveness  of  future  events.  Representatives  of  countries  mentioned 
using  seminar  scenario  data  to  inform  planning  for  future  exercises  and  training  (Estonia  and 
Poland),  anticipate  future  challenges  (Lithuania),  request  additional  information  (Denmark), 
and  facilitate  future  multilateral  dialogue  and  relationship  building.  An  Estonian  Air  Force 
general  participating  in  the  seminar  even  offered  the  use  of  an  Estonian  air  base  to  NATO 
allies  for  training  purposes. 

Participants  from  Sweden  remarked  that  the  discussions  during  this  seminar  were  more 
open  and  candid  than  in  previous  seminars,  and  participants  from  Denmark  thought  that 
engaging  the  involvement  of  even  higher-level  personnel  in  the  seminars  would  be  beneficial. 
Both  comments  highlight  the  value  of  the  seminars  for  building  multilateral  relationships. 
Furthermore,  USA  F  broadened  its  understanding  of  how  to  deter  and  counter  future  hostile 
influence  operations  and  how  to  provide  support  in  such  circumstances  to  its  Baltic  partners. 
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Agenda 

•  Elements  of  the  assessment  framework 

•  Assessing  the  BP  Seminars 

•  Options  for  improving  the  BP  Seminars 
-Process  and  implementation 
-Outcome  and  impact 
-Conclusions 


Finally,  we  look  at  ways  to  substantially  improve  future  BP  Seminars  by  examining  the  process 
and  implementation  of  the  seminars  and  their  outcomes  and  effects.  We  will  conclude  with 
some  specific  recommendations  for  the  USAF,  particularly  for  AF/A3/5. 
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Improving  the  BP  Seminars: 
Process  and  Implementation 

•  Consider  institutionalizing  the  BP  Seminars  with  authoritative 
documentation 

•  Establish  clearer  linkages  to  COCOM  guidance  and  measurable 
objectives  for  each  event 

•  Consider  ways  to  reduce  costs 

•  Follow  up  with  critical  stakeholders  with  an  after-action  report, 
at  a  minimum 


First,  the  process  and  implementation  of  the  BP  Seminars  would  benefit  from  development 
of  authoritative  documentation,  such  as  an  Air  Force  instruction.  As  noted  earlier,  such  docu¬ 
mentation  already  exists  for  similar  programs,  such  as  the  Military  Personnel  Exchange  Pro¬ 
gram  and  the  Joint  Security  Assistance  Training  program. 

Second,  establishing  clearer  linkages  to  combatant  command  guidance  is  important,  as  is 
developing  measurable  objectives  for  the  BP  Seminars. 

Third,  ways  to  reduce  costs  should  be  considered.  For  example,  other  stakeholders  at 
SAF/IA,  the  combatant  and  component  command  levels,  and  possibly  even  the  functional 
MAJCOMs  might  be  able  to  contribute  resources  if  some  of  their  objectives  are  addressed  in 
the  seminars. 

Finally,  developing  an  after-action  report  with  critical  stakeholders  (including  SAF/IA, 
AF/A3/5,  and  the  component  commands)  is  important  for  demonstrating  the  value  of  the 
seminars  and  discussing  any  issues  or  challenges  that  arise  from  them,  using  elements  of  the 
assessment  framework. 
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Improving  Seminar  Outcomes  and  Effects 

Establish  a  process  for  event  follow-up 

•  Postevent  interviews  help  establish  baseline  information 

-  For  example,  partner  nation  plans  to  do  x  (output),  then  follow  up 
by  asking  "did  they  do  x?  Why  or  why  not?"  (outcome) 

•  Administering  a  survey  after  the  event  could  serve  two  purposes: 

-  Survey  could  be  the  primary  tool  to  collect  data  from  the 
delegations  if  time  does  not  allow  during  the  event  itself 

-  Survey  would  show  partners  that  we  want  their  formal  feedback 
after  the  event 

•  Consider  allowing  a  limited  number  of  action  items 

•  Identify  linkages  to  other  A5X-managed  programs  (e.g..  Operator 

Engagement  Talks  and  Air  and  Space  Interoperability  Council) 


Another  way  to  improve  the  outcomes  and  effects  of  the  BP  Seminars  would  be  to  establish  an 
event  follow-up  process. 

One  element  of  this  could  be  to  collect  data  during  the  seminar  specifically  on  a  partner 
country’s  stated  plans  to  implement,  update,  or  change  some  program  element  in  response  to 
the  seminar.  Follow-up  interviews  after  the  event  would  seek  to  find  out  whether  these  plans 
were  successful  and  why  (or  why  not). 

Another  element  could  be  a  post-event  survey  that  would  serve  two  purposes:  (1)  demon¬ 
strate  that  the  United  States  values  the  partner  country’s  formal  feedback  and  (2)  be  a  primary 
data  collection  tool  if  circumstances  do  not  allow  greater  data  collection  by  other  means,  such 
as  rapporteur  note  taking  and  participant  interviewing. 

Limiting  the  number  of  action  items  makes  it  easier  to  address  the  most  pertinent  and 
timely  issues  of  importance  to  the  USAF  and  its  partners. 

Finally,  it  would  be  useful  to  identify  any  potential  linkages  to  other  AF/A5X-managed 
programs,  such  as  the  Operator  Engagement  Talks  and  the  Air  and  Space  Interoperability 
Council,  to  further  engage  partner  countries. 
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Improving  the  BP  Seminars: 
Conclusions  and  Recommendations 


•  Assessing  USAF  BP  programs  is  important  and  not  very  difficult  once  a 
process  is  in  place  and  stakeholders  understand  their  roles 

•  Discussions  across  AF/A5X  on  assessments  of  prior  events  will  have 
ancillary  benefits,  such  as  increasing  visibility  of  other  activities 

•  It  is  essential  to  have  a  mission  statement  and  measureable  objectives 
for  the  program  and  the  individual  events  to  conduct  the  assessment 

•  Corresponding  indicators  should  link  to  each  measureable  objective 

•  Any  follow-up  with  participating  nations  will  most  likely  have  to  be 
spurred  by  the  United  States 
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We  conclude  this  documented  briefing  with  five  recommendations.  First,  assessing  the  BP 
Seminar  program  is  important,  worth  the  effort,  and  not  very  difficult  once  a  process  is  in 
place  and  once  the  various  stakeholders  understand  their  roles. 

Second,  discussions  across  AF/A5X  on  the  assessments  of  prior  BP  events  will  have  many 
ancillary  benefits,  including  increasing  the  visibility  of  other  activities. 

Third,  measurable  objectives  are  essential  for  proper  assessment  of  the  success  and  utility 
of  individual  seminars  and  of  the  seminars  over  time,  respectively.  Furthermore,  development 
of  a  mission  statement  unifies  effort  by  providing  common  purpose  through  addressing  links 
to  strategic  guidance  and  highlighting  the  unique  value  of  the  BP  Seminars. 

Fourth,  corresponding  indicators  should  be  associated  with  each  measurable  objective, 
such  as  need  for  program,  cost  effectiveness,  design  and  theory,  and  outcomes  and  effects. 
In  this  way,  indicators  captured  during  a  BP  seminar  can  be  used  to  make  a  more  informed 
assessment  about  whether  or  not  a  particular  objective  was  accomplished  and  if  not  how  to 
make  future  improvements. 

Finally,  it  would  be  helpful  to  follow  up  with  the  host  country  after  the  event  to  assess 
the  overall  utility  of  the  seminar  and  tailor  follow-on  seminars  to  meet  specific  and  evolving 
host  country  needs.  In  this  way,  the  United  States  will  be  able  to  more  effectively  use  the  BP 
Seminars  to  benefit  the  host  country  and  increase  interest  for  follow-on  seminars. 

In  assessing  the  two  BP  Seminars,  we  believe  the  study  has  provided  a  proof  of  concept 
for  future  assessment  of  USAF-managed  partnership-building  programs  such  as  the  Operator 
Engagement  Talks  and  the  Military  Personnel  Exchange  Program  using  RAND’s  assessment 
framework. 
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Sample  Survey  for  BP  Seminar  Follow  Up 

After-Action  Survey 

UE  Baltic  Seminar,  22-24  September  2009,  Stockholm,  Sweden 

1.  Do  you  have  •  better  understanding  of  different  nations'  perspectives  on  the  High  North  and  other  issues  as  a  result  of  the 
seminar? 

2.  How  useful  were  the  presentations? 

Not  Useful  Useful  Very  Useful 

Presentation  1: 

Presentation  2“ 

Presentation  3: 

Presentation  4: 

Presentation  S: 

3.  How  could  the  presentations  be  made  more  useful? 

4.  Would  additional  subject  matter  experts  have  helped  facilitate  the  discussion? 

5.  Did  you  find  the  seminar's  focus  on  the  High  North  relevant  and  Interesting? 

6.  What  topics  would  you  suggest  for  future  UE  seminars? 

7.  How  did  you  find  the  following; 

Poor  Satisfactory  Excellent 

Travel  arrangements 
Conference  facility 
Meals  and  snacks 
Other  administrative  details? 

8.  Are  you  more  likely  to  work  with  other  nations  on  High  North  issues  in  the  future  as  a  result  of  the  seminar?  (Ask  them  to 
be  specific,  if  possible.) 

9.  Did  you  learn  anything  new  at  the  event  that  can  be  applied  to  your  current  position?  (Ask  them  to  be  spedfk.  If  possible.) 

10.  Would  you  attend  a  future  UE  seminar  or  encourage  others  to  attend? 

11.  Do  you  intend  to  follow  up  with  any  new  people  you  met  at  this  event?  (Ask  them  to  be  specific,  if  possible) 
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Provided  here  is  a  sample  follow-up  survey  for  the  end  of  a  BP  Seminar.  This  survey  solicits 
participant  input  on  a  number  of  seminar  aspects,  including  whether  the  event  enhanced  their 
understanding  of  common  challenges  and  other  countries’  perspectives  on  them,  the  quality 
of  the  presentations,  the  relevance  of  the  topic  and  gathered  expertise,  the  likelihood  that  they 
will  implement  any  knowledge  gained,  and  whether  or  not  they  intend  to  follow  up  with  any 
of  the  other  participants.  All  these  questions  seek  an  understanding  of  outcomes  and  effects  of 
the  seminars  toward  enhancing  future  events. 


Bibliography 


Air  Force  Instructionl6-107,  Military  Personnel  Exchange  Program  (MPEP),  Washington,  D.C.:  Department  of 
the  Air  Force,  February  2,  2006. 

Air  Force  Instruction  (Interservice)  16-105,  Joint  Security  Cooperation  Education  and  Training,  January  3, 

2011.  As  of  July  1,  2011: 

http://armypubs.army.mil/epubs/pdf/R12_15.pdf 

Department  of  Defense  Directive  5230.20,  Visits  and  Assignments  of  Foreign  Nationals. 

Federation  of  American  Scientists,  “Eagle  Vision,  Eagle  Vision  II,”  Washington,  D.C.,  November  21,  1998.  As 
of  December  14,  2010: 

http://www.fas.org/irp/program/process/eagle_vision.htm 

Moroney,  Jennifer  D.  R,  Kim  Cragin,  Eric  Gons,  Beth  Grill,  John  E.  Peters,  and  Rachel  M.  Swanger, 
International  Cooperation  with  Partner  Air  Forces,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-790-AF, 
2009a.  As  of  December  15,  2010: 
http://www.rand.org/pubs/monographs/MG790.html 

Moroney,  Jennifer  D.  R,  Joe  Flogler,  Jefferson  R  Marquis,  Christopher  Paul,  John  E.  Peters,  and  Beth  Grill, 
Developing  an  Assessment  Framework  for  U.S.  Air  Force  Building  Partnerships  Programs,  Santa  Monica,  Calif.: 
RAND  Corporation,  MG-868-AF,  2010.  As  of  December  15,  2010: 
http://www.rand.org/pubs/monographs/MG868.html 

Moroney,  Jennifer  D.  P.,  Jefferson  P.  Marquis,  Cathryn  Quantic  Thurston,  and  Gregory  F.  Treverton,  A 
Framework  to  Assess  Programs  for  Btiilding  Partnerships,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-863- 
OSD,  2009b.  As  of  December  15,  2010: 
http://www.rand.org/pubs/monographs/MG863.html 

Nordic  Council,  “Countries  and  Territories,”  Norden  website,  undated.  As  of  December  14,  2010: 
http://www.norden.org/en/about-nordic-co-operation/countries-and-territories 

Office  of  the  Deputy  Under  Secretary  of  the  Air  Force,  International  Affairs  (SAF/IA),  “U.S.  Air  Force  Global 
Partnership  Strategy:  Building  Partnership  Capability  and  Capacity  for  the  21st  Century,”  December  18, 

2008. 

Office  of  the  Secretary  of  Defense  (OSD),  Guidance  for  Employment  of  the  Force,  Washington,  D.C.: 
Department  of  Defense,  2008,  not  available  to  the  general  public 

Rudner,  Martin,  “Flunters  and  Gatherers:  The  Intelligence  Coalition  Against  Islamic  Terrorism,”  International 
Journal  of  Intelligence  and  Counterintelligence,  Vol.  17,  No.  2,  2004.  pp.  193-230. 

U.S.  Code,  Title  10,  Armed  Forces,  2009. 

- ,  Title  22,  Foreign  Relations  and  Intercourse,  2009. 


35 


